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CODING OF IMAGE SEQUENCES WITH A PLURALITY OF IMAGE BLOCKS AND 
REFERENCE IMAGES 

The invention relates to a method for coding a sequence of digitized 
images with a plurality of image blocks as well as a corresponding 
5 decoding method. The invention also relates to corresponding coding 
and decoding devices. 

Actual video coding standards (for example, see document [1]) allow 
the coding of image sequences, in which case the macro image blocks 
used for an estimation of movement are updated by means of an intra- 

10 coding mode. As a result, errors are not reproduced in the image 

sequence. Updating by means of intra-coding modes can be carried out 
at regular intervals or based on predetermined criteria. For actual 
video coding methods it is possible to use intra-coding modes that 
refer back to several previously coded reference images. However, 

15 there are no mechanisms that allow an efficient video coding with 
inter-coding modes and intra-updating modes over error-prone 
networks. 

It is thus the object of the invention to prepare a method for 
coding a sequence of digitized images that uses a plurality of 
20 intra-coding and inter-coding modes as well as a plurality of 

reference images, and guarantees a reliable reconstruction of the 
digitized images in error-prone networks. 



25 



This object of the invention is achieved according to the features 
of the independent patent claims. Further developments of the 
invention are produced by the dependent claims. 
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The coding method according to the invention codes a sequence of 
digitized images with a plurality of image blocks, in which case the 
image blocks in a section of the image are coded in a first intra- 
coding mode depending on predetermined criteria. In addition, the 
5 image blocks in a section of the image are coded in a second intra- 
coding mode or in an inter-coding mode in which case in the inter- 
ceding mode for the image blocks, an estimation of movement is 
carried out with respect to the reference image blocks of reference 
images of the sequences which are to be coded. Whilst the estimation 

10 of movement according to the method of the invention is being 

performed, only reference image blocks are used which have not been 
coded after the coding of the reference images in a first intra- 
coding mode. This prevents reference being made in the case of the 
inter-coding mode to the reference image areas that are subjected at 

15 least partially to an intra-coding mode. If the coding in the first 
intra-coding mode is carried out particularly for reasons of error 
robustness in order to avoid the reproduction of errors in the case 
of incorrect transmissions, this ensures that the coding is not 
based on image areas that were transmitted incorrectly. Therefore, 

20 an efficient, and at the same time error-robust coding, is 
guaranteed in error-prone networks. 

In one embodiment of the invention, the coding is carried out in a 
first intra-coding mode at regular time intervals. Alternately, the 
coding in the first intra-coding mode can be repeated at random time 
25 intervals. 

In a further embodiment, the coding is carried out in a second 
intra-coding mode or in an inter-coding mode for reasons of coding 
efficiency. For reasons of coding efficiency, an intra-coding mode 
is particularly taken into consideration if an object in the image 
30 sequence only appears temporarily in some images. 
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In a preferred embodiment of the invention, for all the image blocks 
coded in a first coding procedure in an intra-coding mode before a 
second coding procedure is performed, the coding mode to be used for 
coding the image blocks is selected from a number of coding modes. 
5 In this way, an optimum coding mode can be determined. 

In order to determine the optimum coding mode, rate distortion 
criteria can particularly be determined in which case the second 
intra-coding mode or the inter-coding mode is selected depending on 
the rate distortion criteria. Use of rate distortion criteria is 
10 already sufficiently well-known from the prior art and for example 
described in documents [3] and [4]. 

In a preferred embodiment of the invention, the rate distortion 
criteria are determined depending on an error rate to be expected 
when transmitting the coded images. In this case, the distortion of 

15 the pixel values of the images is calculated particularly in order 
to determine these criteria. The distortion of the pixel values 
preferably contains the total of the quadratic differences between 
the pixel values before coding and the correspondingly decoded pixel 
values. Because the distortion is usually not known during coding, 

20 it is possible to estimate the distortion in a particularly 
preferred embodiment. 

In addition to the coding method described above, the invention 
relates to a corresponding method for decoding a sequence of 
digitized images, in which case the method is embodied in such a way 
25 that a sequence of digitized images coded with the coding method 

according to the invention is decoded. In a preferred embodiment of 
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the decoding method, error concealment is used for decoding. 

The invention relates to both the above-mentioned coding and 
decoding method and to a combined method for coding and decoding 
digitized images, in which case the above-described coding and 
5 decoding methods are combined in this method. The invention also 
relates to a device for coding a sequence of digitized images in 
which case the device is embodied in such a way that the coding 
method described above according to the invention can be carried 
out. The invention also includes a corresponding device for decoding 
10 digitized images, in which case the device is embodied in such a way 
that the decoding method described above can be carried out. 

The invention also relates to a combined device for coding and 
decoding digitized images that includes the above-mentioned devices 
for coding and decoding digitized images. 

15 Embodiments of the invention are explained below on the basis of the 
accompanying drawings . 

The drawings are as follows 

Fig. 1 the section of a sequence of decoded images, in which 



20 



case the images were previously coded with a method 
according to the prior art; and 



Fig. 2 



a section of a sequence of decoded images, in which case 
the images were previously coded with the method 
according to the invention corresponding to Figure 1. 



WO 03/063504 



7DE03/00176 



5 

The image sequence shown in Figure 1 was coded with the encoder 
described in document [1] in which case this encoder carries out 
intra-updating modes in an intra-coding mode at regular intervals in 
order to avoid errors from being reproduced in the case of an 
5 incorrect transmission of the image sequence in the decoder. The 
intra-updating modes correspond to the coding modes in a first 
intra-coding mode according to the terminology of the claims. 

The image sequence is transmitted via the Internet test pattern that 
is described in the document [2] . In this case, the image sequence 

10 is transmitted in data packets, where a data packet consists of two 
rows of image blocks. In the text below, image blocks are referred 
to as macro image blocks whose shifting in the case of the inter- 
coding mode is determined by means of movement vectors. The coding 
method, by means of which the image sequence shown in Figure 1 was 

15 coded, also includes a second intra-coding mode and an inter-coding 
mode in the sense of the terminology of the claims. In the inter- 
coding mode, an estimation of movement with respect to a maximum of 
five reference image blocks is carried out. 

The section of the image sequence shows images No. 9 to No . 12 of 
20 this sequence. In order to improve the display of the image 

sequence, a simple error concealment by means of gray shades was 
also used. When the image sequence was transmitted, one packet was 
lost in the first image of the sequence. This transmission error is 
still displayed in image No. 9 of the sequence as can be seen by 
25 means of the horizontal lines in image No. 9 of Figure 1. In image 
No. 10 an intra-updating mode of a section of the image blocks is 
carried out so that a section of the incorrect image are in image 
No. 10 has disappeared. In image No. 11 an inter-coding mode, by 
means of reference images, was carried out in which case the 
30 reference images lie temporally in front of image No. 10 and 

therefore do not include the intra-updating mode. Thus, a large part 
of the incorrect area again appears in image No. 11. The same 
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phenomenon again appears in image No. 12. By means of this 
phenomenon, not only is the distortion in the image increased 
objectively, but the effect in the image is also found to be very 
disturbing from a subjective viewpoint. 

The image disturbances described above can be ascribed to the fact 
that for the coding used in the sequence of Figure 1, a first intra- 
coding mode is connected to an inter-coding mode that uses multiple 
reference images. The occurrence of these disturbances could be 
avoided by not returning to multiple reference images in the case of 
incorrect transmissions, but this would considerably reduce the 
compression efficiency . 

In order to avoid the disturbances described above to the greatest 
possible extent, the coding method according to the invention limits 
the reference images to the effect that for the inter-coding mode 
only such reference image blocks are used as are not subject to any 
intra-updating mode after the reference image has been coded. The 
results of the method according to the invention are shown in Figure 
2. Figure 2 shows the same image sequence as in Figure 1 with the 
difference that the coding method according to the invention was 
used. It is clear that the image disturbances in images No. 11 and 
No. 12 have disappeared. This is due to the fact that for the inter- 
coding mode, no reference images that are transmitted incorrectly to 
the decoder are used. The increase in the bit rate that results from 
the method according to the invention is relatively moderate and is 
only some 5% . 

Exemplary embodiments of the method according to the invention are 
described in detail below. For an embodiment of the method for each 
macro block coding mode m is selected from the number of possible 
inter-coding modes M p and for each reference image r from the number 
of accessible reference images R and optimum movement vectors v(m, 
r) from the number of movement vectors V(m) for the movement 
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compensation. The selection takes place according to the rate 
distortion criteria. Mathematically, the rate distortion criteria 
are displayed as follows: 



(m, r) = arg min(DDrD(m, r, v) +■ X*otioJWor>(m, r, v)) (D 



veV(m) 



5 in which case D DFD (m, r, v) is the distortion according to the 

movement compensation and R mo tion(m^ v) contains the number of bits 
that are needed for coding the specific movement vector. The 
function ((D DFD (m, r, v) +A mot ionRmotion (m, r, v) ) is a so-called 
Lagrange cost function that contains the Lagrange multiplier A motion . 
10 This function is minimized whereby optimum movement vectors are 

determined regarding the distortion and the memory space requirement 
for the movement vector. Therefore, as a first result, optimized 
movement vectors v(m, r) are obtained for each reference image r and 
for each macro block coding mode m. 

15 In a next step, the number of movement vectors is limited by 

removing combinations from the number consisting of the inter-coding 
modes M p and the reference images R, in which referencing takes place 
from image areas that are subjected to an intra-updating mode at a 
later stage, for example, for reasons of error robustness. In this 

20 way, a number 0 P of possible values m and r is obtained for the 
movement vectors and this is as follows: 



0> = {(/w,r)e {M,,R}s^A(v(m 9 rlf,k)>r}, 



(2) 



in which case 



k = 1, K is the number of an image block; 
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f the vector {fi, . .., f K } in which case the variable f A is the digit 
that gives the number of the reference image for the i-th image 
block for which the last intra-updating mode was carried out; 

Sminfi (v (m, r ) , f , k) is an operation that determines the number of the 
reference image for the image block k depending on v(m,r) and f 
that, on the basis of the reference image limitation, is the last 
permitted reference image. 

Should the number of the last permitted reference image exceed the 
number of the reference image r, then it consists of a combination 
(m, r) whose reference image is within the number of reference 
images limited by the method according to the invention. Should the 
last permitted reference image be less than the reference image r, 
then the corresponding combination (m, r) will be rejected. 

The limited set O p of reference images and inter-coding modes m 
resulting from the previous step is combined with a set of intra- 
coding modes M x that can be used in the method according to the 
invention and the optimized coding mode 0(k) is again determined 
from the set union 0 = {M lf 0 P } for each macro block k by means of 
the rate distortion criteria. If this macro block is intra-coded of 
necessity, for example, by regular or random intra-updating modes 
then the number of 0 is limited to only intra-modes, that is 0 = Mi. 
Of course, in this case it is also not necessary to determine O p . 
Mathematically, the rate distortion criteria can again be formulated 
as the minimizing problem of a Lagrange cost function: 



o(k) = arg min (D(o) + AmJ^(o)) , 



(3) 
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in which case R(o), describes the number of bits, to encodes the 
image block in the coding mode o, and D(o) represents the distortion 
for this coding mode. 

If in the method according to the invention there is a regular or 
random intra-updating mode, the distortion is produced as the sum of 
the quadratic differences between the original image block and the 
image block received after the decoding. If the intra-updating mode 
is to be carried out on the basis of an error-optimized channel 
adaptive coding described further below, the distortion is given as 
the expected value of the distortion at the decoder. 

In a next step, it is still necessary to establish whether or not an 
intra-coded image block was intra-coded because of error robustness 
reasons in order to avoid the reproduction of errors or for reasons 
of coding efficiency. An intra-coding mode for reasons of coding 
efficiency particularly prevails if an object in the image sequence 
only appears temporarily. For an intra-coding mode because of coding 
efficiency reasons, a reference image limitation is not desired. In 
order to determine the reasons for the intra-coding mode, a rate 
distortion optimization is again carried out according to equation 
(3) , but where the total number 0 = {M lf 0 P } is always used and as 
the distortion measurement the total of the quadratic differences 
between the original image block and the image block received after 
decoding. The result of the optimization is designated as 6(k). 
Subsequently, an error robustness flag e k is set in which case e k = 
5o(k)*6(k) and 5 CO ndition is the Kronecker symbol that is 1 if the 
condition has been met and otherwise has the value 0. Therefore, the 
intra-coding mode was performed for reasons of error robustness if 
the flag is set to 1 . 



If all the image blocks of an image were processed, the vector f is 
updated for all entries f k for which the error robustness flag e k is 
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set to 1. This avoids a reference image limitation for such intra- 
coding modes which were performed for reasons of coding efficiency 
and thus the appearance and disappearance of objects can be 
efficiently executed by means of coding with the aid of a number of 
reference images. 

An embodiment of the method according to the invention is described 
below in which a channel-adaptive reference image is selected on the 
basis of rate distortion criteria. For this, the distortion D(o) has 
to be estimated in the decoder. Possibilities of estimating this 
distortion are known for example from documents [5] , [6] and [7] . 
One possibility of determining the distortion is the incorporation 
of the random channel behavior C when the distortion is estimated. 
After an image n has been transmitted, the channel behavior C is in 
this case given by means of the binary sequence {0, l} p{n) in which 
case p(n) is the number of packets to be transmitted that are needed 
to transmit the images 1 to n. In this case, a 0 in the sequence 
designates a correctly received packet whereas a 1 indicates a lost 
packet. The random variable that describes the binary sequence up to 
image n is designated as C P(n ) - The pixel distortion in the decoder 
depends on the pixel value reconstructed in the decoder that is 
designated as s^and which is unknown to the encoder carrying out the 
coding. The pixel distortion depends on the channel behavior C and 
the selected coding mode o, i.e. §i = Si (C p{n ),o). The distortion is 
estimated as the total of all the expected values of the quadratic 
pixel distortions dj.(o) of all the macro blocks i in which case it is 
assumed that the channel behavior C p(n) is known to the encoder. The 
pixel distortion di(o) for the macro block i is as follows: 
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in which case, E Cp {n-i) represents the expected value of the quadratic 
difference of the original pixel value and the reconstructed pixel 
value averaged over the channel C p(n -i) . 

In order to calculate the expected value, the following easy method 
5 can be used. It is assumed that T copies of the random variables 
^channel behavior" are available in the encoder. These copies are 
designated as Cp (n) (t) , with t = 1, T. It is also assumed that 

all the random variables C P{n) (t) are distributed independently, 
identically and statistically. Therefore, according to the strict 
10 law for high numbers, T -> 00 is as follows: 



»T] A - S*(C,w(t\oP = JScx-i Sr-StfCrt^o) 2 = d(o) , 



(5) 



Therefore, with the expression on the left side, the expected value 
di(o) can be estimated and in a next step, the expected distortion 
Di(o) calculated. The reconstruction of the pixel values depends on 
15 the channel behavior C p < n -i) (t) as well as the concealment in the 

decoder. By means of the last-mentioned formula it is possible to 
estimate in the encoder the intensity of the distortion in the 
decoder . 
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Patent claims 

1. Method for coding a sequence of digitized images with a 
plurality of image blocks, wherein: 

5 - the image blocks in a section of the image are coded in a 

first intra-coding mode depending on predetermined criteria; 

the image blocks in a section of the image are coded in a 
second intra-coding mode or in an inter-coding mode in which 
case, in the inter-coding mode for the image blocks, an 
10 estimation of movement is carried out with respect to the 

reference image blocks of reference images of the sequences 
which are to be coded; 

whilst the estimation of movement according to the method of 
the invention is being performed only reference image blocks 
15 are used which have not been coded after the coding of the 

reference images in a first intra-coding mode. 

2. Method according to Claim 1, characterized in that the 
predetermined criteria for carrying out the coding in a first 
intra-coding mode are error robustness criteria with respect to 

20 an incorrect transmission of coded images. 

3. Method according to Claim 1 or 2, characterized in that the 
coding is carried out in a first intra-coding mode at regular 
time intervals. 

4. Method according to Claim 1 or 2, characterized in that the 
25 coding is carried out in a first intra-coding mode at random 

time intervals . 



5. 



Method according to one of the preceding claims, characterized 
in that the coding in a second intra-coding mode or in an inter- 
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coding mode is carried out for reasons of coding efficiency. 

6. Method according to one of the preceding claims, characterized 
in that for all the image blocks coded in a first coding 
procedure in an intra-coding mode before carrying out a second 

5 coding procedure, the coding mode to be used for coding the 

image blocks is selected from a number of coding modes . 

7. Method according to Claim 6, characterized in that the rate 
distortion criteria are determined and the selection of a second 
intra-coding mode or the inter-coding mode depends on the rate 

10 distortion criteria. 

8. Method according to one of the preceding claims, characterized 
in that the rate distortion criteria are determined as a 
function of an error rate to be expected when the coded images 
are transmitted. 

15 9. Method according to Claim 7 or 8, characterized in that, the 
distortion of the pixel values of the images is calculated to 
determine the rate distortion criteria. 

10. Method according to Claim 9, characterized in that the 
distortion of the pixel values contains the total of the 

20 quadratic differences between the pixel values before coding and 

the correspondingly decoded pixel values. 

11. Method according to Claim 9 or 10, characterized in that the 
distortion is estimated. 

12. Method for decoding a sequence of digitized images in which case 
25 the method is embodied in such a way that a sequence of 

digitized images is decoded with a method from the preceding 
claims . 
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13. Method according to Claim 12, characterized in that an error 
concealment is carried out in the decoded images. 

14. Method for coding and decoding digitized images including a 
method according to one of the claims 1 to 11 and a method 
according to Claim 12 or 13. 

15. Device for coding a sequence of digitized images, characterized 
in that the device is embodied in such a way that a method can 
be carried out according to one of " the Claims 1 to 11. 

16. Device for decoding digitized images, characterized in that the 
device is embodied in such a way that a method can be carried 
out according to Claim 12 or 13. 



17. 



Device for coding and decoding digitized images including a 
device according to claim 14 and a device according to Claim 16. 



